Due to rapidly evolving network attacks, detection of malicious/unusual traffic patterns in enterprise traffic has become vitally important. As opposed to traditional signature-based schemes that detect known attacks, Intrusion Detection Systems (IDSs) that detect deviation from normal traffic profiles of users, hosts and networks are now becoming increasingly popular. The possibility that the peculiar behavior could be suspicious/malicious is the intuitive rationale that necessitates the need of an IDS. Such detection system comprises of a model characterizing the normal behavior of network traffic. It is then decided as to what percentage of deviant activity must be set as an anomalistic threshold.
Any effective IDS will have to somehow learn a good classification threshold for an arbitrary benign behavior in real-time. To make matters worse, raw data that are input to an IDS typically show considerable variations. Traffic characteristics vary considerably across organizations, network deployment points, and diurnal and other usage patterns. Similarly, host-based anomaly detection metrics are a function of user behavior, applications being used, operating system, hardware, etc. As input data characteristics vary, determination of a fixed threshold requires repeated manual intervention. In a typical operational scenario, a system/network administrator is responsible for adjusting the sensitivity of a network-based intrusion detector when the number of false alarms (i.e., traffic classified as malicious but which is in fact benign) increases. Similarly, host-based IDSs expect a user to adjust its sensitivity to cater for his/her security and behavioral requirements. Such repeated manual input renders an IDS less automated and more prone to configuration errors. Moreover, in a real-time system it is difficult to ascertain if a manually-configured threshold is yielding good accuracy.
In A. Lakhina et al., “Mining anomalies using traffic feature distributions” ACM SIGCOMM, 2005, J. Jung et al., “Fast portscan detection using sequential hypothesis testing”, IEEE SSP, 2004 and K. L. Ingham et al., “Comparing anomaly detection techniques for http, RAID, 2007, incorporated herein by reference authors propose some network anomaly detectors which provide methods to calculate optimum thresholds for their specific algorithms. However, most of these studies do not cater for the time varying behaviour of the input and consequently fail to provide acceptable performance under varying traffic conditions. M. Agosta, C. D. Wasser, J. Chandrashekar and C. Livadas, “An adaptive anomaly detector for worm detection”, Usenix SysML, 2007 incorporated herein by reference, proposed an anomaly detector which adjusts its threshold according to the variations observed in input. However, no such generic technique is available that can cater for varying input and work with any intrusion detector. Since intrusion detection algorithms are regularly updated in response to evolving attack characteristics, it is reasonable to assert that a practical threshold adaptation technique, instead of being devised for a specific intrusion detection algorithm, should seamlessly operate with any given algorithm. However, a generic adaptive threshold tuning technique that can automatically achieve a suitable operating point on the ROC plane for any existing real-time anomaly detector is not available.
The present invention proposes an IDS which automatically detects varying input data patterns and adjusts its classification threshold accordingly. Such an adaptive thresholding mechanism will enable an IDS to achieve good operational points on the ROC plane. As a by-product, adaptive thresholding also eliminates the need for human threshold tuning, thereby making an IDS more automated. The invention and its embodiments are discussed in detail in the description section of the present disclosure.